Explicit and Persistent Knowledge in Engineering Drawing Analysis

Domain knowledge permeates all aspects of the engineering drawing analysis process, including understanding the physical processes operating on the medium (i.e., paper), the image analysis techniques, and the interpretation semantics of the structural layout and contents of the drawing. Additionally, an understanding of the broader reverse engineering context, within which the drawing analysis takes place, should be exploited. Thus as part of a wider project on the reverse engineering of legacy systems, we have developed an agent-based engineering analysis system called NDAS (nonDeterministic Agent System). In this paper, we discuss the nature of such a system and how knowledge can be made explicit (both for agents and humans) and how performance models can be de£ned, calibrated, monitored, and improved over time through the use of persistent knowledge. A framework is proposed that allows computational agents to: (1) explore the threshold space for an optimal analysis of the drawing, (2) control information gain through agent invocation, (3) incorporate and communicate knowledge, and (4) inform the software engineering and system development with deep knowledge of the relationships between modules and their parameters (at least in a statistical sense). Explicit and Persistent Knowledge in Engineering Drawing Analysis Thomas C. Henderson School of Computing, University of Utah Abstract Domain knowledge permeates all aspects of the engineering drawing analysis process, including understanding the physical processes operating on the medium (i.e., paper), the image analysis techniques, and the interpretation semantics of the structural layout and contents of the drawing. Additionally, an understanding of the broader reverse engineering context, within which the drawing analysis takes place, should be exploited. Thus as part of a wider project on the reverse engineering of legacy systems, we have been developing an agent-based engineering analysis system called NDAS (NonDeterministic Agent System).Domain knowledge permeates all aspects of the engineering drawing analysis process, including understanding the physical processes operating on the medium (i.e., paper), the image analysis techniques, and the interpretation semantics of the structural layout and contents of the drawing. Additionally, an understanding of the broader reverse engineering context, within which the drawing analysis takes place, should be exploited. Thus as part of a wider project on the reverse engineering of legacy systems, we have been developing an agent-based engineering analysis system called NDAS (NonDeterministic Agent System). In this paper, we discuss the nature of such a system and how knowledge can be made explicit (both for agents and humans) and how performance models can be defined, calibrated, monitored, and improved over time through the use of persistent knowledge. A framework is proposed that allows computational agents to: (1) explore the threshold space for an optimal analysis of a drawing, (2) control information gain through agent invocation, (3) incorporate and communicate knowledge, and (4) inform the software engineering and system development with deep knowledge of the relationships between modules and their parameter (at least in a statistical sense). 1.0 Introduction The reverse engineering of legacy systems is a difficult and complex problem, but of vital importance. This usually involves a physical instance of the system, as well as some paper drawings produced by hand or from mechanical CAD systems. The goal may range from exact replication, to changing some parameters, to a major re-design. For example, Figure 1 shows a gearbox that operated for many years as part of a shipyard crane system. Developing reverse engineering techniques from such a physical example and any available related engineering drawings is our goal. Figure 1. Newport News Gearbox Figure 2 shows the overall reverse engineering system we are developing; the goal is to take advantage of data about the system in all its forms: drawings, 3D scans, and CAD models as they are constructed, as well, and to allow the user virtual access during the redesign process (see Figure 3). The wider knowledge involved includes manufacturing information and constraints, design analysis codes (e.g., stress or aerodynamics), cost/performance models, etc. Figure 2. Reverse Engineering System Figure 3. Envisioned Virtual Interface to model surface, point cloud and drawing data. 2.0 The Vision Before giving details on the systems we have been building, we would like to give our vision of how to construct a system so that domain knowledge can be exploited in a powerful way. We now give a high-level summary of our proposed theoretical framework and enumerate some advantages that may result from this approach. Figure 4 shows a set of agents, Ai, each of which produces various outputs using a set of parameters and thresholds, Ti, and each having an associated model (or set of models), Pi(Xi|Ti), describing the agent’s variance from the ideal in terms of some appropriate measure. Knowledge of three sorts (physical, image analysis, and structural interpretation) is available and informs the agents’ actions and understanding of each others results. Higher-level control processes may exploit this in several ways: 1. Explore the threshold space for global optima (see feedback loop in Figure 4). 2. Control acquisition of new data (e.g., view token generation as state estimation and select agent action that optimizes information gain). 3. Incorporate knowledge in abstract form and communicate abstractions between agents and users. 4. Inform the software engineering and system development with deep knowledge of the relationships between modules and their parameters (at least in a statistical sense). Figure 14. Smart Agents Network System The current status of the project (called the Smart Agent Network System or SANS) is that the core image and structural analysis components have been developed and applied to engineering drawing analysis to gain experience and insight into crucial agents, their parameters and interactions. We are now exploring the representation of this domain knowledge in specific nomenclatures. We are also investigating state estimation frameworks to provide a more incremental analysis based on observations provided by the system, and the associated information measures (see [Catlin 1989] for an introduction to the area). Notice that each program execution can be viewed itself as a measurement on the image, and the set of measurements will be used by a control process to achieve the best interpretation of the drawing. Note that state estimation is a reasonably mature tool in many engineering applications. More recently, such methods have been incorporated in multisensor systems to try and achieve optimal control and sensing [Durrant-Whyte 2003]. More broadly, this approach is starting to see proponents in scientific computing as well [Emery 2001, Emery 2002]. We believe that it can be applied to general large software systems; however, in this paper, we discuss how it might be used in engineering drawing analysis. 2.0 Engineering Drawing Analysis with NDAS We have shown that a structural model may be realized through a set of software agents acting independently and in parallel to ultimately achieve a coherent analysis of CAD drawings [Henderson2003a,Henderson2003b,Henderson2003c,Swaminathan2002]. The high-level goals of the analysis are to: • Understand legacy drawings. • Acquire context of field and engineering data. • Respond to external analysis, user input. • Integrate drawing analysis in redesign. NDAS allows multiple agents to produce the same type of data, for example, line segments or text. Other agents which use these entities as inputs may choose from any or all of the available sets of data to produce their own data. Moreover, even a single agent can produce its output using multiple thresholds, or can be asked by another agent to produce output with a given set of control parameters. This allows people or more sophisticated agents to explore the entire parameter space of all the agents involved in the analysis. The mechanism to handle the combinatorial explosion of data is tied to the structural definition of the engineering drawing, and uses syntactic analysis to eliminate redundant comparisons. This symbolic redundancy calculation uses both the syntax of structural rewrite rules, as well as parsing constraints on the tokens generated from the image analysis to achieve orders of magnitude reductions in the possible combinations of tokens. However, NDAS to date has done little else to incorporate or exploit the wealth of other knowledge involved in understanding engineering drawings. 3.0 Knowledge about Engineering Drawing Analysis Figure 4 shows the sequence of paper drawing creation and exploitation with which we are concerned. We consider knowledge about physical processes, image analysis and document interpretation. Figure 4. Engineering Drawing Analysis Process 3.1 Physical Processes It is important to capture knowledge about all aspects of the physical processes involved. For example, printing gives rise to certain errors that can influence the image analysis and subsequent interpretation. During storage and usage, it is possible to introduce lines by folding or creasing, or to obscure lines and text by stains, writing or damage to the paper. Scanning is itself a physical process subject to motion blur, lighting, scale and other perturbations. Good understanding is necessary for robust and correct analysis, and a good synthesis model will allow the controlled creation of test data with defects. 3.2 Image Analysis Discrete geometry plays a large role in the analysis of engineering drawings, and involves abstract notions, including: • 0-dimensional objects: isolated points, corners, branch points, end points, etc. and relations: distance, near, same kind, etc. • 1-dimensional objects: line segments, straight segments, circles, boxes, etc. and relations: collinear, parallel, perpendicular, neighbor, closed, etc. • 2-dimensional objects: blobs (e.g., arrowheads) and relations: above, left of, touches, occludes, etc. Moreover, these notions cannot be implemented perfectly, and it is important to know how the realizations differ from the ideal (e.g., what’s the threshold for parallel?). Even more important is the relation of these notions and their recovered approximations to the semantic tokens which form the basis for the structural analysis. 3.3 Structural Analysis The structure of the drawing is given by a set of tokens (e.g., line segments, text, pointers, graphics, manufacturing symbols, etc.) and the relations that hold between them. Thus, the production of the tokens is crucial, and interpretation problems arise when tokens are missing, broken into parts, or falsely reported. The relations between the tokens need to be clearly defined, as well as the amount of divergence from the ideal. Context of various sorts is also extremely important, and ranges from geometric frame (which way is up?) to drawing type (detail drawing, assembly description, manufacturing constraint requirements, etc.). These various sets of knowledge are usually not made explicit, either during the development of the system or for exploitation during an analysis. We are interested in answering the following kinds of questions: 1. How can this knowledge be made explicit? 2. How can the differences between the ideal and the implementations be given? 3. Can some of the knowledge (ideal or performance) be learned by the agents? 4. How can people interact with this knowledge to understand why the system does something or to change how the system does it? 5. How can the knowledge be exploited during the analysis of one image; over a set of related images; over various projects, i.e., in order to gain and record more insight on engineering drawing analysis in the log term. It is essential to answer these questions so that the system can improve over time, and be more effectively understood and exploited by its human operators. 4.0 Proposed Method We propose the following approach to address this problem: 1. Give a specification for the ideal. 2. Give ways that implementation can differ from ideal. 3. Give a measure of the difference. 4. For every analysis, keep a record of the ideal referent, actual produced, difference measure and analysis parameters. For example, parallel segments should ideally have 0 degrees difference in angle. A difference measure would be the actual difference in angle, or some monotonically increasing function (square, exponential, etc.). Various implementations would carry different information; e.g., if parallel is computed from the two segment angles, then an angle difference threshold would be kept; if parallel is determined by whether the points defining the one segment are all the same distance from the other segment, then the maximum and minimum distances would be kept. It is possible to have agents for both parallel operators, and the system can decide (based on training or operator feedback) which is better. This goes with our notion to develop a system which allows many different analysis methods in parallel, and from this wealth of data, chooses between them to construct the best interpretation possible. This approach also fits well with statistical approaches. For example, various information measures can be defined and used to steer the analysis. Once we have established mechanisms for knowledge expression and use, we will explore alternative mechanisms for the exploitation of that knowledge (for example, Durrant-Whyte and colleagues [Durrant-Whyte 2003] have developed methods to maximize information gain with each observation action – this approach might give good results here). 4.1 Knowledge about Engineering DrawingsLet’s look in more detail at the knowledge that would be useful in this application. Asfor engineering drawings per se, Table 1 gives some of the useful information: SubjectIssuesForm of KnowledgeLayoutUp/down, text orientation Semantic network/ grammarSymbolsAlphabet, digits, special Dictionary; images; netsReferencesConventions for pointers,names, use of circles, etc.Semantic net; imagefeaturesCharactersLanguage, numbers,measuresSemantic nets, featurevectors, imagesReal world semantics Manufacturing info, 3D, 2Dprojections, etc.Semantic network Table 1. Types of Knowledge in Engineering Drawing Analysis As can be seen, most of this knowledge, if it exists, might be better expressed as asemantic network or in vector or image form. We are currently investigating theconstruction of a domain ontology, and hope to base it on the Standard Upper MergedOntology (or SUMO) [Niles 2001]. In this way, we make the assumptions of the agentsexplicit, and provide a SUO-KIF [SUO-KIF] interface to other users and systems.However, it must be pointed out that our domain requires analogical forms of knowledgeas well, including: images, 3D data sets from Coordinate Measurement Machines or laserscanners, etc. Some axiomatizations and ontologies for geometry exist (e.g., see [Asher1995, Pratt 1997, and Tarski 1956], but their usefulness in this context remains to beseen. Image analysis has its own set of concerns, including: • 1D segments,• pixels (digitization),• relations, and• realization of geometry. Algorithms include: thresholding foreground/background, thinning, segment extraction,straight segment determination, geometric objects detection (e.g., boxes, circles), pointerdetection, and text detection. Each of these must deal with thresholds, sensitivityanalysis, quality estimates, complexity, and robustness with respect to other algorithms. Finally, knowledge about goals may influence agent actions; here are some goals that thesystem may be asked to achieve: • Find part name.• Find label information.• Extract references to other parts.• Get dimension information for specific part features.• Determine manufacturing constraints.• Determine safety or other special descriptions in the text. These various forms of knowledge should not be static, but should be adjustable overtime, as more experience is gained. For example, the use of pointers in drawings can bequite creative, and these need to be cataloged and accounted for. At a minimum,threshold exploration should be possible and recorded. Another issue is what needs to be communicated between agents (and/or users) whichincludes at least the following: • the goal,• the results of an agent; this includes the info produced, info about the productionof the info, and some quality of result measures, and• feedback to an agent; for example, “this data resulted in no solution” or “parallelconstraint needs to be tighter” or “your results are not necessary for this goal”;this last feedback would lead to greater efficiency if agents know when they areunnecessary. For example, the circle agent uses simple 1D segments (a set of pixels) as input andchecks if the set of pixels forms a circle. However, this agent is not necessary for theanalysis of the title block of a drawing; it is essential, however, for full drawing analysis.The result of the analysis is a list of point sets determined to constitute circles, and foreach circle gives the center and radius, the segments or pixels involved, a quality measureof the circle, and the resources used to produce the circle (e.g., data files used, space andtime complexity, etc.). It may also be necessary to include information about why thethresholds and parameters were selected. As an example of feedback that the circle agentmay want to provide, suppose that it uses straight line segments to detect circles (i.e., aset of straight line segments form a circle if they are connected end to end and theirpoints do not lie to far from a circle); if the straight segments are fit too coarsely, theymay not form a circle, when in fact the pixel data would permit a circle. Thus, the circleagent may want to ask the segment agent to re-fit the data with a tighter linear fitthreshold. As a starting point, we have investigated the knowledge about thresholds and theirinterplay between entities produced, consumed, and the semantic tokens generated.Figure 5 shows the image analysis part of NDAS. Threshold utilization is indicated bythe circled numbers. Table 2 gives the meanings of the thresholds. Figure 5. The Image Analysis Agents and the flow of data between them. Circle no. in Figure 5 Thresholds/parameters Related Impact1foreground/background extra/missing pixels;connectivity of segments2pixel curvature parameters corner detection, straightsegment endpoints3circle fit parameterscircle detection, referencedetection4line fit parametersnumber and quality ofsegments5collinear; line fit parameters large-scale object detection6endpoint distances; segmentlengths; collinearpointer ray detection,dimension analysis,references 7segment length, separationthreshold, parallel,perpendicular, duplicatethresholdbox detection; documentblock analysis, text analysis Table 2. Image analysis agent thresholds and parameters and their impact. We term the image analysis knowledge given in Figure 5 as superficial knowledge, sinceit concerns only the external relations between the agents and their products. Thus,information about the organization of modules, which use the data from which others,their production information, the quality measures on the data, the amount and trends ofdata production, and the system activity all fall under this term. Opposed to that is deep knowledge, which concerns the inner workings and decisionrationales for implementations, threshold settings, etc. This then includes an idealdescription of the process, an explanation set of how the implementation differs from theideal, a characterization of the likelihood of the variances from the ideal, and the relationof the variations to further processing, including semantic token (terminal symbol)creation and semantic analysis. To clarify these ideas, let’s consider the image thinning process. There is a mathematicalnotion of a valid thinning operator on point sets, but implementations may vary from thisideal for many reasons and with different implications. Consider the four versions of thethinned partial segment in Figure 6. Figure 6. Four variations of a thinning operation. Which of these is produced may significantly impact later analysis; e.g., abstractionsbased on end point, branch point and straight line segment relations can be radicallydifferent. Figure 7 shows a set of relation graphs for the thinned objects above. Figure 7. Graphs of connectivity between end points (e1,e2,...), and branch points(b1,b2,...) of thinned objects from Figure 6. (Between every pair of nodes is a, notnecessarily straight, line segment.) As can be seen, the number of line segments, the position of their endpoints and thegeometric relations between them (distance, parallel, etc.) can all be greatly affected bythese differences in the thinning. Thus, what might be viewed as a local or minoralgorithm issue, may lead to a radical change in performance (including increase incomplexity if lots of small segments are generated) if there is no knowledge of how oneprocess impacts other processes through shared analysis objects. It is of great interest tounderstand these relationships, and to declare them when the system is designed andimplemented, but even if that is not possible or accurate (the developers may notunderstand the impact!), it would be good to allow the system to determine some of thisknowledge as various algorithms are executed with different parameter values. In terms of the thinning operation, we might proceed as follows: Ideal definition of thinning: One example of this is the medial axis transform [Blum1973]. This is the set of points such that a circle centered at the point touches theboundary of the object in at least two distinct places. Algorithm difference from ideal: the algorithm may approximate the ideal definition inorder to reduce computational complexity and because the ideal notions don’t applyperfectly to digital geometry. The following differences may occur: 1. Ends of segments may be fragmented.2. Corner regions of segment may be fragmented3. Medial axis may be displaced from actual corner location. Measures of difference: Several possibilities exist to measure the three differences listedabove. There are two levels of measure, however. First, it is of interest to measureindividual errors in terms of the number of extra segments produced, or the distance athinned set is displaced from a point of interest in the original point set. In addition, it isuseful to have some statistics over the whole population. For example, this might beeither (1) a likelihood on the number of extra fragments expressed as a mean andvariance or in other forms, or as a function of the original segments, the features of thesegment or those of the thinned segment. For example, if the thinned segment isperfectly straight, then it is most likely that it perfectly represents the ideal.